Wireless immersive simulation system

ABSTRACT

Computer-based systems provide a wireless, high fidelity, real-time video and audio immersive environment for multiple users. A user may be equipped with a head mounted display device which wirelessly receives real-time video signals transmitted by a radio transceiver linked to a computer that generates the real-time video images. Aspects include body, arm and leg motion and orientation sensors in wireless communication with a radio transceiver linked with a computer. Video images may be generated using a portable array of user dedicated computers controlled by a single computer that is controlled by an operator.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Divisional of U.S. patent application Ser.No. 13/948,992, filed Jul. 23, 2013, entitled, “WIRELESS IMMERSIVESIMULATION SYSTEM,”, which is a non-provisional of U.S. PatentApplication No. 61/674,698, filed Jul. 23, 2012, entitled, “WIRELESSIMMERSIVE SIMULATION SYSTEM,” all of which are hereby incorporated byreference in their entirety for all purposes.

BACKGROUND

Training for hazardous tasks, such as combat or firefighting, is oftendifficult or expensive to perform physically on training grounds. Therecould be numerous situations with very different physical surroundings,which would be expensive or time-consuming to create. Monitoringtrainees' responses would also add difficulty and expense.

Training for such situations using simulated environments allows forgreater safety in training, less expense in creating physical stages ontraining grounds, and much greater flexibility in the type and number oftraining scenarios presented to trainees. Simulated training is improvedwhen an immersive experience is presented to the trainee. In immersivetraining, the trainee virtually experiences all aspects and sensationsof the hazardous situation. Simulated or virtual immersive trainingsituations often require a trainee to wear a portable computer attachedby cabling or wires to computers controlled by operators or trainers. Atrainee often wears a specialized vision helmet having a head-up displayin front of the trainee's face to present virtual images of the trainingscenario. A trainee often wears a specialized suit with motion capturedevices attached. The vision helmet and motion capture suit are alsooften attached by cabling to the trainers' computers. Such cabling isoften used to provide the data rates need for high fidelity real-timevideo for presentation on the head-up display and to receive responses,both verbal and motion, from the trainee.

However, such specialized equipment attached to the trainee cancompromise the verisimilitude of the simulation since the portablevision computer, vision helmet and motion capture suit present differentweight and shape characteristics than the equipment that would actuallybe used by the trainee. For example, the vision helmet could be verydifferent in size, shape or weight from a helmet worn by a soldier or bya firefighter. Further, cabling necessary for communications with thetrainer's computers can greatly restrict the range of motion of thetrainee.

So there is a need for ways to provide more realistic immersivetraining. Such training would allow the trainee to wear the equipmentexpected to be used for the situation. The additional monitoring andtraining equipment would be minimally intrusive and still providerealistic real-time video and audio communications with the trainee. Inthe case of team training, it would be advantageous for the trainingequipment to allow real-time communications between trainees in theteam.

BRIEF SUMMARY

This Summary does not in any way limit the scope of the claimed subjectmatter.

In an embodiment, a system for providing a simulated immersive trainingenvironment is disclosed. The system may enable one or more trainees orusers to experience computer-generated visual and/or auditory sensationsof a situation in order to learn how to respond. The system may compriseat least one computer subsystem capable of generating a real-time videoimage sequence, a first radio transmitter/receiver communicativelylinked with the computer subsystem and capable of transmitting thereal-time video image sequence, a directional antenna communicativelylinked with the at least one radio transmitter/receiver subsystem, andat least one Head-Mounted Display (HMD) device configured to be worn ona user and comprising a second radio transmitter/receiver capable ofreceiving the real-time video image sequence transmitted by the firstradio transmitter/receiver and of displaying the real-time video imagesequence on at least one screen positioned before the eyes of the user.

Additional and/or alternative embodiments may comprise any combinationof the following elements. There may be a second antenna distinct fromthe directional antenna and communicatively linked with the first radiotransmitter/receiver. The first radio transmitter/receiver may becapable of simultaneously transmitting a first radio signal on a firstradio channel using the directional antenna and transmitting and/orreceiving a second radio signal on a second radio channel, distinct fromthe first channel, using the second antenna. The real-time video imagesequence may be transmitted on the first radio signal. The real-timevideo image sequence may comprise a 3-dimensional stereo image and beprojected on dual screens of the HMD. The real-time video image sequencemay be transmitted by the first radio transmitter/receiver using the 60GHz frequency bands. The computer subsystem may be capable of generatinga plurality of real-time video image sequences simultaneously. Thecomputer subsystem may comprise a controller computer and a plurality ofuser-dedicated computers.

Additional and/or alternative embodiments of the system may comprise anycombination of the following elements. There may be a user hand-heldtraining device comprising a sensor that transmits information about thehand-held training device on the second radio channel. There may beuser-mounted sensors which measure and transmit orientation informationto the first radio transmitter/receiver using the second radio channel.The system may also comprise user-mounted microphone and/or speakers fortransmitting and receiving audio communications on the second radiochannel.

In another embodiment, a second system is disclosed for receiving awirelessly transmitted simulated immersive training environment. Thesecond system may comprise a HMD to be worn by a user and that includesa real-time video image sequence display device, a battery powered radiotransmitter/receiver capable of receiving a real-time video imagesequence on a first radio channel, and capable of simultaneouslytransmitting and/or receiving radio signals on a second radio channeldistinct from the first channel, and an orientation sensor linked to theHMD. The HMD transmits orientation data on the second radio channel.

Additional or alternative embodiments of the second system may compriseany combination of the following elements or aspects. The HMD may beconfigured to be attached to the exterior of the surface of a helmet.The orientation sensor may be battery powered, with the battery beingrechargeable from an inductive charger. The first radio channel may bein the 60 GHz band.

In another embodiment, a third system is disclosed for transmittingsensor data from a user within a simulated training environment,comprising: a battery powered orientation sensor attachable to the user,the orientation sensor comprising a radio transmitter/receiver; a hubradio transmitter/receiver and a system radio transmitter/receiver. Theorientation sensor transmits an orientation measurement using the sensorradio transmitter/receiver to the hub radio transmitter receiver andreceives signals from the hub radio transmitter/receiver. The hubtransmitter/receiver transmits signals to the orientation sensor,receives the orientation measurement transmitted from the orientationsensor, and transmits the orientation measurement to the system radiotransmitter/receiver. Additional and/or alternative embodiments maycomprise any combination of the following aspects. The hubtransmitter/receiver is battery powered. An orientation sensor may alsodetect motion and/or position. The hub radio transmitter/receiverattaches to a device held by the user during a simulation.

BRIEF DESCRIPTION OF THE DRAWINGS

A further understanding of the nature and advantages of variousembodiments may be realized by reference to the following figures. Inthe appended figures, similar components or features may have the samereference label. Further, various components of the same type may bedistinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

FIG. 1 illustrates a system providing real-time wireless video and audioimmersive training to users.

FIG. 2 illustrates a user equipped with wireless sensors and a videodisplay device.

FIG. 3 illustrates a portable carrier for computers and a radiotransceiver.

FIG. 4 is a cross-sectional diagram of a display device attached to auser's helmet.

FIG. 5 illustrates a configuration of a directional antenna.

FIG. 6 is a diagram of a first sensor patch.

FIG. 7 is a diagram of a second sensor patch.

FIG. 8 is a diagram of a sensor-transmitter.

FIG. 9 is a diagram of a second sensor used on a hand-held device.

FIG. 10 is a diagram of an wireless architecture for a simulationenvironment.

FIG. 11 is a diagram of sensors communicatively linked to a hubtransmitter/receiver.

FIG. 12 is a diagram of a computer architecture.

DETAILED DESCRIPTION

For the purposes of explanation, the ensuing description providesspecific details in order to provide a thorough understanding of variousembodiments. It will be apparent, however, to one skilled in the artthat various embodiments may be practiced without some of these specificdetails. For example, circuits, systems, networks, processes, and othercomponents may be shown as components in block diagram form in order notto obscure the embodiments in unnecessary detail. In other instances,known circuits, processes, algorithms, structures, and techniques may beshown without unnecessary detail in order to avoid obscuring theembodiments. In other instances, well-known structures and devices areshown in block diagram form.

The embodiments disclosed in the following description relate tosystems, methods and devices by which realistic immersive simulatedtraining experiences can be provided to users or trainees by wirelesscommunications. The embodiments may provide individual immersiveenvironments to more than one user at a time to provide training forcooperative group actions. Immersive simulated training is advantageousas it eliminates the need for constructing physical trainingenvironments, and a great variety of training scenarios can be providedby varying the programming that creates the simulations. Immersivetraining may allow users or trainees to experience the training scenariowithout being distracted by actual physical surroundings, such as atraining room, that are not part of the simulated training scenario.

Applications include training a soldier or squad of soldiers for streetpatrols. However, it will be apparent to one of skill in the art thatother applications are possible. Alternate applications include trainingfirefighters for entering buildings, and police officers for approachingdangerous situations.

The immersive training may be provided by having each user wear ahead-mounted display (HMD) device that displays real-time video of thetraining scenario on a screen or screens in front of the users face tomake the user feel as if he or she is part of the simulated environment.Immersive training may also comprise audio speakers to provide thesounds of the simulated environment. In some embodiments the system, orjust the HMD itself, may be completely immersive, in that it prevents auser's ability to receive sights or sounds not provided by the system.In this way the user may more likely become completely mentally engagedin the simulation, and so experience more realistic training.

Training simulations may be created on a computer or computer systemsand transmitted and regenerated on the HMD, or other devices, thatprovide the immersive environment to the user or users. The visualand/or auditory aspects of a training scenario may be generated muchmore quickly and cheaply using computer graphics software than actuallyconstructing the physical environment. For example, for training a squadof soldiers, visual images of a street and surrounding buildings can besimulated so soldiers can experience patrolling the street. Withcomputer graphics software, the aspects of the street, such as streetwidth and height of surrounding buildings, can quickly be changed toalter the training scenario.

The computer or computer systems may generate entire video sequences ofa simulated environment so that a user may view actions within thesimulated environment in real-time. Real-time video simulations arethose that present successive still images to a user at a ratesufficient for the user to experience no visual delays between imagesand to experience viewing the images as if physically viewing an actualenvironment. The computer or computer system may generate videosequences to simulate motion of the user or users through the simulatedenvironment.

Immersive simulated training is more effective in the case that actionsby a user are accounted for when generating and presenting the simulatedenvironment to the user. For example, how a soldier moves down a streetor where he is looking influences the sounds and visual images thatshould be presented to the soldier through the HMD and/or other devices.For the computer system to adjust the simulated environment, thecomputer system must have a way to receive information about the user'sactions. In various embodiments, a user has motion and/or orientationsensors attached to various body parts, such as arms, legs, torso andhead. The sensors are communicatively linked to the computer system sothe computer system can generate the appropriate video image sequence,and other aspects of the simulated environment, to correspond to theposition, viewing direction and orientation of the user.

Immersive simulated training is more effective when the user wears theequipment that would be worn if actually in the simulated environment.For example, a soldier might wear a military helmet, body armor, combatboots, a standard pack, and possibly other equipment. A firefightermight wear fireproof garments, oxygen tank, helmet and possibly otherequipment. Further, in realistic situations, it is unlikely that asoldier or firefighter would be attached to electronic cabling or wires,or carry a heavy computer system. Thus it is advantageous for animmersive simulated training system to avoid so encumbering the user. Invarious embodiments disclosed herein, this advantage, among others, isachieved by means of a HMD which receives real-time video imagesequences and other aspects of the simulated environment by wirelessradio communications transmitted from the a computer system thatgenerates the simulated environment. By separately generating thesimulated environment and wirelessly transmitting it to the HMD, theuser is enabled to wear the actual equipment that would be worn in thesimulated environment. A further advantage to wireless transmission tothe HMD of the user is that the computer system used for generating thesimulated environment can be separately located and unconstrained insize, and thus comprise much more computing and simulation capabilitiesthan would be possible if the user had to carry a computer forgenerating and rendering the video or other aspects of the simulatedenvironment.

To achieve greater verisimilitude, in various embodiments sensorsattached to a user are disclosed which are designed to fit unobtrusivelyon or in parts of the equipment that a user might normally use in anactual situation corresponding to the simulation. The sensors may alsocomprise wireless transmitters and/or receivers so the sensorinformation can be conveyed to the computer system for adjustment of thesimulation.

Further details of the aspects of the embodiments discussed above arenow provided with reference the figures.

FIG. 1 illustrates an example of system 100 implementing an immersivesimulated training environment, in which a squad of trainee soldiersexperiences a training scenario controlled by an operator 103. Thesimulated environment of the training scenario is shown being generatedby computer systems 104 a and 104 b, and is shown being transmitted tothe soldiers from radio transmitter/receiver systems 106 a and 106 b.The users, i.e. the trainee soldiers in the example shown, the operator,the computer system and radio transmitter/receiver systems are locatedin a training area 101. In various embodiments, the training area may belocated in a single building or open outdoor area. The training area maybe shielded from electromagnetic or audiotory interference. In theexample shown in FIG. 1, the users are organized into three groups inseparate locations 102 a, 102 b and 102 c that are in differentdirections from the radio transmitter/receiver systems 106 a and 106 b.As explained more fully below, a respective directional transmission toeach group 112 a, 112 b and 112 c may be performed by the radiotransmitter/receiver systems.

Examples of a training scenario for soldiers could be a simulated streetto be patrolled, the inside of building in which rooms must be searched,or a forest to be traversed. In other examples, the users could befirefighters, and the training scenario could involve a burning house.

In the example shown in FIG. 1, some or all of the users may be equippedwith individual head-mounted display (HMD) devices. As described morefully below, an HMD may project real-time video image sequences ondisplay screens mounted in front of the user, and isolate the user fromsights or sounds not generated for the training scenario. Some or all ofthe users may have individual speakers for hearing sounds generated forthe training scenario. Some or all of the users may have microphones fortransmitting voice responses as part of the training scenario.

In FIG. 1, the training scenario being presented to the users isgenerated by computer systems 104 a and 104 b. As explained more fullybelow, the computer systems may comprise one or more computers, and maygenerate a real-time video image sequence corresponding for each of theusers. The computer systems may also generate sound signals appropriatefor the training scenario being simulated. The computer systems may alsoexecute the software necessary to have the video image sequence and/orother information transmitted to the users.

The computer systems are communicatively linked to radiotransmitter/receiver systems 106 a and 106 b in order for the video andother aspects of the training scenario to be transmitted to the HMD andother devices associated to each user. In the embodiments shown in FIG.1 the video signals may be transmitted to particular subgroups of usersby directional transmission. One advantage of directional transmissionto just a subgroup of users is that less power can used. Anotheradvantage is the ability to use the same frequency for transmission todifferent subgroups of users. Also, by using multiple computer systemseach linked with a corresponding radio transmitter/receiver system usingdirectional transmission to just a subgroup of users, the computersystems can use the computational capabilities more efficiently togenerate more realistic video images or other aspects of the trainingscenario.

Directional transmission may be accomplished using one or moredirectional antennas on each radio transmitter/receiver system, such asa dish antenna, or by beamforming or beam steering from an antennaarray, or by another method as will be apparent to one of ordinary skillin the art. FIG. 1 schematically shows directional antennas 110 a, 110 band 110 c. Though shown as whip antennas, as stated above thedirectional antennas could be implemented using other antennageometries. Further, though shown physically integrated with the radiotransmitter/receiver systems in FIG. 1, in some embodiments an antennamay be a separate portable device that is linked with a radiotransmitter/receiver system by cabling or wiring at the training area101 as the need arises. An exemplary embodiment of such a separateantenna is explained below in relation to FIG. 4.

The radio transmitter/receiver systems may also be capable ofsimultaneous transmission using one or more auxiliary channels otherthan the channels used for transmitting real-time video image sequencesof the training scenario. An advantage would be to allow use of amultiple access communication protocol such as WiFi on an auxiliarychannel for voice signaling to and from the users, but reserving thevideo image channel for transmitting the high data rates needed forreal-time video transmissions. In FIG. 1 the use of an auxiliarytransmission channel is indicated diagrammatically by the second antenna108 attached to the radio transmitter/receiver systems 106 a.

It will be apparent to one of skill in the art that the number of eachtype of antenna communicatively linked with a radio transmitter/receiversystem may be varied according to need.

The particular numbers of operators, users, computer systems, trainingareas, radio transmitter/receiver systems and antennas shown in FIG. 1are exemplary only. As will be apparent to one of skill in the art, thenumber of each may be varied as needed. The embodiments may varysubstantially in form and functionality, depending on the desiredapplication.

FIG. 2 illustrates an example configuration of various devices that maybe attached to a user to implement an immersive simulated trainingenvironment. The computer system 104 a and a radio transmitter/receiversystem 106 a as described above are in communication with a user. Thecommunication is illustrated occurring by transmissions from thedirectional antenna 110 a of a real-time video image sequence signal 112and additional antenna 108 for communications to and/or from a radiotransmitter/receiver attached to the user.

As described in detail below, the user may have attached to helmet a HMD202 and a display screen 204 to receive and display respectively areceived video image sequence.

To provide information on the user's responses to the simulation, theuser may have attached one or more sensors. The sensors may be able todetect or deduce any combination of motion, orientation or position. Thesensors may use gyroscopes, such as ring gyroscopes, accelerometers, orother equipment known to one of skill in the art. As is known to one ofskill in the art, position and/or velocity can be inferred byintegration of acceleration. Various embodiments of the sensors aredescribed in detail in reference to FIGS. 6, 7, 8, and 9.

Sensor 206 illustrates an orientation and/or motion sensor mounted on aweapon held by the user during the simulation to provide feedbacksignals to the computer system 104 a about how and where the user ispointing the weapon. In other applications, the sensor 206 may bemounted on a different hand-held device. For example, sensor 206 couldbe mounted a simulation of a firefighter's hose.

Sensor 208 is an orientation and/or motion sensor mounted to the body ofthe user. Sensor 210 is a leg mounted orientation and/or motion sensorthat may be worn inside a pant leg pocket, or attached to the user's legwith straps, as detailed below. There may be one or more such sensor foreach leg. Sensor 212 may be an orientation and/or motion sensor attachedto the arm of the user. There may be one or more such sensor for eacharm.

FIG. 3 is a block diagram 300 of a portable rack-mounted computer systemand radio transmitter/receiver system, according to one embodiment. Anadvantage of the embodiment is that the necessary computer system andradio transmitter/receiver system are portable to allow immersivetraining simulations to be performed in variety of locations. Also,components can be added, upgraded or replaced as needed.

The radio transmitter/receiver system 302 shown in FIG. 3 includes awireless access point (WAP) and a router. In the embodiment shown, theWAP/router performs the coordinating functions for a wireless local areanetwork (WLAN). The WLAN may be implemented with any of variety ofcommunication protocols, such as WiFi, a version of the IEEE 802.11standards, a cellular telephone link, or an AM/FM link. Otherembodiments of a radio transmitter/receiver system may be used, as knownto one of skill in the art. As used herein, a radio transmitter/receiversystem may include any combination of the following: a radio transmitterfor sending radio signals to an antenna, a modulator for converting abaseband signal to a radio signal for transmission, a radio receiver forreceiving a radio signal from an antenna, a tuner for selecting aparticular radio channel to receive, a demodulator for converting areceived radio signal to a baseband signal, amplifiers and filters formodifying electronic or radio signals, and other components as would beapparent to one of skill in the art. A radio transmitter/receiver systemmay be implemented as a single unit or as separate units.

In one embodiment, the radio transmitter/receiver system 302 implementsa WLAN for receiving sensor data from sensors on one or more users. Thesystem 302 may also be capable of receiving voice data from one or moreusers. The system 302 may also be capable of transmitting voice or datato one or more users.

The system rack 300 also may include computer monitoring and entrycomponents 304 such as a keyboard, a monitor, and/or a computer mouse.Other components will be apparent to one of skill in the art. Thecomponents 304 may also contain a switch to allow an operator toalternate which computer on the rack is being the components connect to.

There may be one or more user computers, 306 a, 306 b, 306 c and 306 d,on the rack. In one embodiment, a User Computer is dedicated togenerating the simulated training environment for one respective user. AUser Computer may execute computer graphics software to generatereal-time video image sequences of the simulated environment appropriatefor the respective user. The images generated may be based oninformation about the positions and orientations of sensors attached invarious locations on the user, or on an implement held by the user.

The information about the positions and orientations of sensors may beprovided to a User Computer by a controller computer 308 from radiosignals transmitted by the sensors and received from the WAP/Router 302.The controller computer may also control the transmission of video imagesequences over the directional antenna.

An operator may interact with the user computers and/or the controllercomputer through a keyboard, mouse and/monitor 304. A simulation may besent to a monitor from a user computer for output to the operator usingan high-definition multimedia interface (HDMI) Repeater 310, and theselection of the user computer selected using a HDMIkeyboard/video/monitor switch 312. The rack may have an powerdistribution unit (PDU) 314 to allow multiple components to be placed onthe rack, and ensure safe operation in case of a fault in a component.

FIG. 4 is an illustration of a directional antenna 400. Otherembodiments may combine, separate, and/or substitute components forthose shown in FIG. 4. A person of ordinary skill in the art willrecognize many variations.

The computer and radio transmitter/receiver system 300 may use more thanone such directional antenna to direct communications with varioussubgroups of users. The directional antenna 400 may be communicativelylinked with at least one system 300 by video-capable cabling 410, orother high data rate connections. The cabling 410 is capable oftransmitting data at rate sufficient for real-time video imagesequences. Non-limiting examples of such cabling include fiber opticcable and coax cable. The video-capable cabling 410 may comprise abundle of individual cables from separate user computers.

The directional antenna 400 may comprise an antenna array 402 on ahorizontal beam. The horizontal beam may be capable of being adjustablein pitch to steer the transmissions. The beam may be capable of beingrotated horizontally with respect to the rest of the directionalantenna.

The horizontal beam maybe positioned on a vertical, extensible shaft404. In some aspects the shaft is fixed in length. The shaft 404 may behollow and the cabling 410 extend inside the shaft's interior 406. Theshaft 404 may be supported collapsible legs 408 that extendnon-vertically from the shaft. A tripod arrangement for the collapsiblelegs 408 is shown in FIG. 4, but alternate embodiments, such as fourlegs, will be apparent to one of skill in the art. In alternateembodiments, the legs 408 supporting the shaft 404 are not collapsible.

In one set of embodiments the directional antenna is used fortransmitting the real-time video image sequences of a training scenariogenerated by the computer system or systems described above. Inadditional and/or alternate embodiments, the directional antenna may beused for receiving signals transmitted from devices mounted on a userwhich are described in further detail below.

FIG. 5 is a cross-sectional illustration of an embodiment of ahead-mounted display (HMD) device 500 and the components therein. In oneembodiment, the HMD is attached to a helmet 502 of a user. At the rear(posterior) end of the helmet is an attachment component 504 of the HMDdevice. The HMD device may also comprise a strap 518 that extends overthe exterior portion of the user's helmet attaches to a front mount 520.The tension in the strap may be adjusted by a tensioner 516.

To supply electrical power to electrical components of the HMD devicewithout the use of wires from a remote source, the HMD device maycomprise a battery 506. The battery may be rechargeable either by aninductive charger, or by a physical connection such as a cable or wirewhen not in use. An advantage of the system is that the battery mayallow a more realistic simulation for the user by avoiding physicallyconstraining wire or cable connections from the HMD device on the userto a separate power supply.

The status of one or more components of the HMD device may be indicatedby one or more light emitting diodes 508. Examples of such a statusinclude on/off, adequate battery voltage or sufficient received radiopower.

The HMD device may include a wireless video antenna and receiverassembly 514. The assembly may be configured to receive real-time videoimage sequences transmitted from the radio transmitter/receiver system300. To achieve the data rate necessary for real-time video reception,the receiver may comprise an antenna and demodulator configured forsignals channels in the 60 GHz band. Examples of 60 GHz band technologyis specified in the IEEE 802.11 ad standards. As signals in the 60 GHzband are not able to go through walls, the system 100, when using 60 GHzband for video transmission, operates in a training area without walls,and with the directional antennas raised high enough to transmitthroughout the training area. In other embodiments, the assembly 514 mayuse another communication technology and frequency bands able totransmit real-time video image sequences without delay. The radiochannel used for the video channel may be dedicated to download only.

The assembly 514 may also comprise a second radio transmitter/receiverable to transmit and receive on a second radio band. For example, thesecond radio transmitter/receiver may implement a version of the 802.11(WiFi) to allow communications within a wireless local area network(WLAN). By communicating over a WLAN transmissions from a user can bereceived at system 300 and processed to update the simulation inreal-time. For example, voice responses of one user can be relayed toother users. Also, data from sensors attached to the user may betransmitted on the WLAN to the computer system 300 and used to modifythe video image sequence created for the user.

The HMD may comprise an internal head tracking sensor 510. The sensormay detect the orientation and/or motion of user's head. This may beaccomplished by use of gyroscopes, such as ring gyroscopes, oraccelerometers. The sensor may comprise electronics to calculate motionand/or position from acceleration data. The head tracking sensor may bebattery powered. In an alternate embodiment, a head tracking sensor maybe an external component that is connected to the HMD at one or moreconnection points 512. The connection may include electricalconnections. The head tracking sensor may be communicatively linked withthe radio transmitter/receiver 514 so that the detected head orientationor motion may be transmitted on the second channel to the computersystem 300.

The HMD may comprise a mounting plane 520 to be located at the front ofthe user's helmet. Attachable to the mounting plane may be one or morevideo display screens that extend downward from the mounting plane to bepositioned immediately in front of the user's eyes so that the user cansee only what is on the display. In some embodiments there may twoscreens capable of displaying stereo images to produce a 3-dimensionalvideo simulation.

The HMD may also comprise a microphone positioned for the user to speakinto. The signals from the microphone may be sent to the radiotransmitter/receiver 514 for transmission to the computer system 300.The HMD may also comprise audio speakers positioned near the user's earsto receive voice and/or audio transmissions related to the simulation.The voice and/or audio signals to be sent to the speakers may bereceived on the radio transmitter/receiver 514.

By using one or more of the elements just described, the HMD can providea user with the experience of being completely or nearly completelyimmersed in the training scenario being simulated. Verisimilitude canalso be enhanced if information about a user's actions, such as voiceresponses and/or changes in position or orientation, are known by thecomputer system 300 so that the simulation, including the video imagesequences for the user, can be adapted. Determining such user actionsmay be determined by using sensors attached to the user, as nowdescribed.

FIG. 6 is an illustration of an embodiment of leg sensor pocket plate600. The leg sensor pocket plate (LSPP) may be designed to be insertedinto a known pocket of the apparel typically worn by a user. Forexample, the LSPP may be designed to fit into an outside leg pocket of asoldier's combat uniform. The LSPP may comprise a flexible mountingsurface 602. In some embodiments the flexible mounting surface may be ofplastic or thin metal sheet so that the LSPP may support a solidmounting component 604, but yet still provide flexibility to fit into apocket and flex to conform to the shape of a leg. The flexible mountingsurface may comprise orientation indicators 606 a and 606 b, which allowthe user to insert the LSPP into a pocket with a correct orientation.

The solid mounting 604 may comprise one or more holes 610 fororientation and motion sensors. The solid mounting may comprise one ormore cutouts 608 a and 608 b to allow for attachment of optional legsensor straps. The straps may be used either in place of, or in additionto, other means of securing the LSPP.

The LSPP may be attachable to a radio transmitter/receiver to transmitorientation or motion data. Alternatively, the LSPP may have an internalbattery powered radio transmitter/receiver. The transmitted data may betransmitted to a radio transmitter/receiver in the HMD, or to a radiotransmitter/receiver that is an element of a user hand-held device, asdescribed below.

FIG. 7 is a diagram of an embodiment of a sensor assembly 700. Mountingtabs 710 a and 710 b allow the sensor assembly to attached to either auser's garments or hand-held device. The mounting tabs are attached to aprotective enclosure 702, which may comprise a material such as plasticor metal, and may be sealed to prevent moisture or contaminates fromentering the enclosure.

Within the protective enclosure may be a circuit card assembly (CCA) towhich may be attached an antenna 706, a microelectromechanical system(MEMS) 708 and a battery 710 to power the MEMS and antenna. The MEMS mayimplement the operations of orientation or motion sensing, andconversion of the sensed data to electrical form for modulation andtransmission over the antenna.

FIG. 8 is diagram of an embodiment of a hub radio transmitter/receiverassembly 800. The hub may allow communication between (1) a plurality ofsensors on a user or a user's hand-held device, and (2) a radiotransmitter/receiver system, such as 106 a and an attached computersystem, such as 104 a, which is directing a simulation. By wirelesslytransmitting data from sensors to a hub radio transmitter/receiver,coordination of sensor data transfer is simplified and interferenceprevented, and lower power radio transmitter/receiver implementationsmay be implemented in the sensors, rather than each sensor using a morepower consuming WiFi technology for data transmission. Also, wires fromthe sensors to the hub are eliminated, allowing a user greater freedomof movement.

A protective enclosure 802 as described previously for the sensorassembly 700, may contain a CCA to which is attached an antenna 808, aWiFi or Bluetooth radio transmitter/receiver module 806, one or moreprocessors 810, a battery 814 to provide power to the components on theCCA and one or buttons 812. One of the buttons may be an on/off button,while others may allow a user to change the operational state ofprocessors, or to enter either programming commands or user responses tothe simulation.

A connector 816 allows a non-wireless sensor to input data to the hub800. In the case that the hub is to be mounted to the weapon, or otherhand-held device of a user, a sensor on the weapon may have only adirect, e.g. wired, connection and so avoid being equipped with a radiotransmitter/receiver. The hub may be attached to a weapon by means of amounting tab 818.

Other embodiments may combine, separate, and/or substitute componentsfor those shown in FIG. 8. A person of ordinary skill in the art willrecognize many variations.

FIG. 9 is a diagram of an embodiment of an alternate sensor assembly900. In one embodiment, assembly 900 attaches, using the mounting tab902, to a weapon held by a user in a simulation. The stock 922 may allowa user to grip and hold a part of a weapon.

Assembly 900 comprises a protective enclosure 904 as describedpreviously. Within the enclosure are a MEMS 906 and a battery 908, bothas described previously in regards to FIG. 7. Also illustrated withinthe protective enclosure may be a wireless joystick CCA 910. Thewireless joystick may include a low power radio transmitter/receiver(e.g. Bluetooth or Zigbee) for transmitting data to the hub radiotransmitter/receiver 800. In an alternate embodiment, the CCA 910 mayconnect with the hub 800 by a physical connector.

The data transferred by the joystick CCA 910 may be entered by the userby the buttons 920 and/or 924. Other embodiments may combine, separate,and/or substitute components for those shown in FIG. 9. A person ofordinary skill in the art will recognize many variations.

FIG. 10 illustrates a block diagram of a wireless video hardwarearchitecture that may implement components of the system 100, asdescribed in reference to the preceding figures.

An equipment rack 1050 may hold the one or more user-dedicated computers1058, as described in relation to FIG. 3. A user-dedicated computer mayalso comprise an image generator and image output (IG/OUT) 1057 thatgenerates real-time video image sequences, an Ethernet connection 1056that allows communication with other computers, and a sound card 1055for generating or relaying sound transmissions. The user-dedicatedcomputer may also include an audio signal connection 1053 and an HDMIconnection 1054 to a controller computer 1052.

The audio component of a simulation may be transmitted on thecommunication link 1082 by an audio repeater 1070 to a user microphone1018. The microphone 1018 may include a radio transmitter/receiver totransmit voice or sensor data from the user on the second radio channeldistinct from the high data rate channel 1080 that is used fortransmitting the real-time video image sequences.

The real-time video image sequence data created by the image generator1057 is relayed through the HDMI link 1054 to the controller computer1050 and then transferred over the HDMI repeater 1060 to a wirelesstransmitter/receiver 1040. The wireless transmitter/receiver 1040 maycomprise a video modem 1046 to convert the video image signal to a highdata rate transmission format. The formatted signal is then modulated bya transceiver 1044 for radio transmission from the antenna 1042, overthe high data rate channel 1080.

A component of an HMD may be a HMD wireless link 1030 to receive thetransmitted signal containing the real-time video image sequence. Thelink 1030 may comprise an antenna assembly 1032, a transceiver 1034capable of converting the radio signal to an electronic format, and avideo modem 1034 for converting the electronic formatted signal intodata for display.

The HMD wireless link 1030 may transmit the real-time video imagesequence data over an HDMI link 1020 to HMD outputs 1010. The outputsmay comprise one or more video display screens 1012 a and 1012 b, andleft and right audio speakers 1014 a and 1014 b. The speakers may alsoreceive inputs from a headset 1016.

Other embodiments may combine, separate, and/or substitute componentsfor those shown in FIG. 10. A person of ordinary skill in the art willrecognize many variations.

FIG. 11 illustrates a block diagram of a wireless sensor radio hardwarearchitecture 1100. There may be a plurality of sensors, which areexemplified by the orientation and/or position sensor 1102 linked withsensor electronics assembly 1109, and motion and/or orientation sensor1110. A sensor may comprise a radio transmitter/receiver system capableof wirelessly transmitting radio signals containing acquired sensor datato a hub radio transmitter/receiver device 1120. As detailed below, thehub radio transmitter/receiver 1120 may comprise a plurality of radiotransmitter/receiver subsystem, exemplified by components 1121, 1122 and1123, for wirelessly communicating with the various sensors. The hubradio transmitter/receiver 1120 may also include a second radiotransmitter/receiver subsystem, exemplified by components 1124 and 1125,for wirelessly communicating over a WLAN with a WAP linked to a computersystem. The hub radio transmitter/receiver may also be directlyconnected to at least one other sensor, 1128, receiving orientation,position or motion data from a user hand-held device.

In one exemplary application, a user is a soldier being trained in asimulated environment. The sensor 1110 is attached the soldier's leg andis capable of detecting any or all of the motion, position ororientation of the soldier's leg. There may be a pair of such sensors,one for each leg. The sensors may be battery powered, and transmit databy a wireless communication system 1110, so that the soldier isunencumbered by wires during the simulation. The soldier may havemounted on his chest or arms another orientation or motion sensor 1102connected to the a radio transmitter/receiver 1109 for wirelesslytransmitting the data about the orientation or motion. There may be morethan one such body sensor, for example there may be a pair of suchsensors, one for each arm, and a sensor attached to the chest. eachsensor may have a corresponding radio transmitter/receiver system, eachof which may use a different frequency for transmission and reception.

In this exemplary application the sensors wirelessly communicate with ahub radio transmitter/receiver 1120. In this application, the hub 1120may be configured as a single physical unit that attaches to a weaponused by the soldier. Being attached to the weapon allows the hub tocommunicate with at least one weapon sensor 1128 mounted on the weapon.Such weapon sensors may detect the orientation and/or motion of theweapon. Such a weapon sensor may be connected by wires to the hub radiotransmitter/receiver.

In this example the hub radio transmitter/receiver may be able totransmit signals to the sensors on the soldier, 1110 and 1109, forexample to coordinate timing of transmission and reception oforientation or other data from the sensors. The hub radiotransmitter/receiver may also communicate using a transmitter/receiver1124, separate from any transmitter/receiver and using a separate radiochannel, for communication with a separately located computer system.The separately located computer system may be generating the video oraudio components of the simulation.

It will be apparent to one of skill in the art that the exemplaryapplication just described is not limiting, and the architecture 1110can be applied in a variety of other applications. Another exemplaryapplication would be when the user is a firefighter, and the handhelddevice is a simulation of a hose. Further details for the componentsshown in 1100 are now presented.

The sensor 1110 may comprise a motion node 1112, which detects motion.Motion may be detected by as described above in relation to FIGS. 6through 9. The motion detector may convert its physical measurement toelectronic form, and the electronic signals may then be processed by theprocessor 1114. Such processing may include any of filtering, analog todigital conversion, and modulation for transmission by radio signaling.

The sensor 1110 may include a transceiver 1116 which accepts anelectrical signal and transmits the signal using the antenna 1118 to ahub transmitter/receiver 1120. In one embodiment, the transmission is bymeans of a short range communication protocol such as Bluetooth orZigbee. The antenna 1118 and transmitter/receiver 1116 may also receivetransmissions from hub 1120, which are demodulated and processed by theprocessor 1114. The received transmissions may include timing and othercontrol commands.

An additional and/or alternate embodiment of sensor is illustrated bythe sensor 1102, which is linked to a processor and transmitter/receiverelectronics unit, 1109. The link may be by a wire or fiber optic cableto allow the physical sensing unit 1102 to be separate from theelectronics unit 1109. This may allow for shielding, and for smallersensor attachments. The electronics unit 1109 may comprise a processor1104 capable of performing the operations described for the processor1114. The electronics unit may also comprise a radiotransmitter/receiver 1106 linked with an antenna 1108 for transmittingand receiving signals from the hub transmitter/receiver 1120, asdescribed above in relation to the transmitter/receiver 1116 and antenna1118.

The hub transmitter/receiver 1120 may comprise a plurality oftransceiving circuits comprising an antenna 1121, atransmitting/receiving circuit 1122, and processor 1123. In oneembodiment there may be one such combination dedicated to communicationwith one corresponding sensor. In an alternate embodiment, there may beonly one combination of antenna/transceiver/processor and thecombination is capable of receiving and transmitting on multiplefrequencies simultaneously. The processor 1123 may be able to performthe operations discussed above in regards to the processor 1114.

The hub transmitter/receiver 1120 may comprise a sensor or sensorreceiver 1128. This may be a motion or orientation sensor that is aphysical part of the hub 1120, or may be a connector, such as jack orwire, by which an external sensor may input data to the hub. The signalsfrom the sensor 1128 may be controlled by processor 1127, which mayfunction as described above.

The hub transmitter/receiver 1120 may comprise memory 1126 on which codefor operation of the various components may be stored. The memory may beimplemented using electrically erasable programmable read only memory(EEPROM) 1126.

The hub transmitter/receiver 1120 may comprise a processor 1125 andtransmitter/receiver 1124 for wirelessly communicating the data receivedfrom the sensors to a remote transmitter/receiver, and for receivinginformation from the remote transmitter/receiver. In one embodiment, thetransmitter/receiver 1124 uses WiFi as part of a WLAN, or a similarmultiple access communication technology.

Many of the various components and devices described in the precedingsections may use a computer or processor based electronics. These can beimplemented in many ways, as now described.

A computer system as illustrated in FIG. 12 may be incorporated as partof the previously described elements of a VTO or as part of the elementsat an operator center. FIG. 12 provides a schematic illustration of oneembodiment of a computer system 1200 that can perform steps of themethods. It should be noted that FIG. 12 is meant only to provide ageneralized illustration of various components, any or all of which maybe utilized as appropriate. FIG. 12, therefore, broadly illustrates howindividual system elements may be implemented in a relatively separatedor relatively more integrated manner.

The computer system 1200 is shown comprising hardware elements that canbe electrically coupled via a bus 1205 (or may otherwise be incommunication, as appropriate). The hardware elements may include one ormore processors 1210, including without limitation one or moregeneral-purpose processors and/or one or more special-purpose processors(such as digital signal processing chips, graphics accelerationprocessors, and/or the like); one or more input devices 1215, which caninclude without limitation any of the input devices previouslydescribed, as well as a mouse, a keyboard, and/or the like; and one ormore output devices 1220, which can include without limitation any ofthe output devices previously mentioned, as well as a display device, aprinter, and/or the like.

The computer system 1200 may further include (and/or be in communicationwith) one or more non-transitory storage devices 1225, which cancomprise, without limitation, local and/or network accessible storage,and/or can include, without limitation, a disk drive, a drive array, anoptical storage device, a solid-state storage device, such as a randomaccess memory (“RAM”), and/or a read-only memory (“ROM”), which can beprogrammable, flash-updateable and/or the like. Such storage devices maybe configured to implement any appropriate data stores, includingwithout limitation, various file systems, database structures, and/orthe like.

The computer system 1200 might also include a communications subsystem1230, which can include without limitation a modem, a network card(wireless or wired), an infrared communication device, a wirelesscommunication device, and/or a chipset (such as a Bluetooth™ device, an802.11 device, a WiFi device, a WiMax device, cellular communicationfacilities, etc.), and/or the like. The communications subsystem 1230may permit data to be exchanged with a network (such as the networkdescribed below, to name one example), other computer systems, and/orany other devices described herein. In many embodiments, the computersystem 1200 will further comprise a working memory 1235, which caninclude a RAM or ROM device, as described above.

The computer system 1200 also can comprise software elements, shown asbeing currently located within the working memory 1235, including anoperating system 1240, device drivers, executable libraries, and/orother code, such as one or more application programs 1245, which maycomprise computer programs provided by various embodiments, and/or maybe designed to implement methods, and/or configure systems, provided byother embodiments, as described herein. Merely by way of example, one ormore procedures described with respect to the systems and methodsdiscussed above might be implemented as code and/or instructionsexecutable by a computer (and/or a processor within a computer); in anaspect, then, such code and/or instructions can be used to configureand/or adapt a general purpose computer (or other device) to perform oneor more operations in accordance with the described methods.

A set of these instructions and/or code might be stored on anon-transitory computer-readable storage medium, such as thenon-transitory storage device(s) 1225 described above. In some cases,the storage medium might be incorporated within a computer system, suchas computer system 1200. In other embodiments, the storage medium mightbe separate from a computer system (e.g., a removable medium, such as acompact disc), and/or provided in an installation package, such that thestorage medium can be used to program, configure, and/or adapt a generalpurpose computer with the instructions/code stored thereon. Theseinstructions might take the form of executable code, which is executableby the computer system 1200 and/or might take the form of source and/orinstallable code, which, upon compilation and/or installation on thecomputer system 1200 (e.g., using any of a variety of generallyavailable compilers, installation programs, compression/decompressionutilities, etc.), then takes the form of executable code.

It will be apparent to those skilled in the art that substantialvariations may be made in accordance with specific requirements. Forexample, customized hardware might also be used, and/or particularelements might be implemented in hardware, software (including portablesoftware, such as applets, etc.), or both. Further, connection to othercomputing devices such as network input/output devices may be employed.

As mentioned above, in one aspect, some embodiments may employ acomputer system (such as the computer system 1200) to perform methods inaccordance with various embodiments of the invention. According to a setof embodiments, some or all of the procedures of such methods areperformed by the computer system 1200 in response to processor 1210executing one or more sequences of one or more instructions (which mightbe incorporated into the operating system 1240 and/or other code, suchas an application program 1245) contained in the working memory 1235.Such instructions may be read into the working memory 1235 from anothercomputer-readable medium, such as one or more of the non-transitorystorage device(s) 1225. Merely by way of example, execution of thesequences of instructions contained in the working memory 1235 mightcause the processor(s) 1210 to perform one or more procedures of themethods described herein.

The terms “machine-readable medium” and “computer-readable medium,” asused herein, refer to any medium that participates in providing datathat causes a machine to operate in a specific fashion. In someembodiments implemented using the computer system 1200, variouscomputer-readable media might be involved in providing instructions/codeto processor(s) 1210 for execution and/or might be used to store and/orcarry such instructions/code. In many implementations, acomputer-readable medium is a physical and/or tangible storage medium.Such a medium may take the form of a non-volatile media or volatilemedia. Non-volatile media include, for example, optical and/or magneticdisks, such as the non-transitory storage device(s) 1225. Volatile mediainclude, without limitation, dynamic memory, such as the working memory1235.

Common forms of physical and/or tangible computer-readable mediainclude, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, or any other magnetic medium, a CD-ROM, any other opticalmedium, punchcards, papertape, any other physical medium with patternsof holes, a RAM, a PROM, EPROM, a FLASH-EPROM, any other memory chip orcartridge, or any other medium from which a computer can readinstructions and/or code.

Various forms of computer-readable media may be involved in carrying oneor more sequences of one or more instructions to the processor(s) 1210for execution. Merely by way of example, the instructions may initiallybe carried on a magnetic disk and/or optical disc of a remote computer.A remote computer might load the instructions into its dynamic memoryand send the instructions as signals over a transmission medium to bereceived and/or executed by the computer system 1200.

The communications subsystem 1230 (and/or components thereof) may be thecommunication component previously discussed, or a separate element ofthe computer system 1200. Generally the communications subsystem 1230will receive signals, and the bus 1205 then might carry the signals(and/or the data, instructions, etc. carried by the signals) to theworking memory 1235, from which the processor(s) 1210 retrieves andexecutes the instructions. The instructions received by the workingmemory 1235 may optionally be stored on a non-transitory storage device1225 either before or after execution by the processor(s) 1210.

It should further be understood that the components of computer system1200 can be distributed across a network. For example, some processingmay be performed in one location using a first processor while otherprocessing may be performed by another processor remote from the firstprocessor. Other components of computer system 1200 may be similarlydistributed.

In the description above and the claims that follow, the word “a” or“an” preceding an element does not exclude the presence of a pluralityof such elements.

What is claimed is:
 1. A system for transmitting sensor measurement datafrom a plurality of users within a simulated training environmentcomprising: a plurality of sensors attachable to the plurality of users,each sensor comprising a battery powered sensor radiotransmitter/receiver and an antenna, wherein each sensor is capable ofmeasuring at least one of orientation, position and motion; at least onehub radio transmitter/receiver; a system radio transmitter/receiver; arespective hub radio transmitter/receiver corresponding to a first userand a second user of the plurality of users, and the respective hubradio transmitter/receivers separately attach to respective devices heldby the first user and the second user during a simulated trainingscenario; and a plurality of sensors attached to the first user, andwherein each of the plurality of sensors attached to the first usertransmits measurement data to the hub radio transmitter/receiver of thefirst user; wherein each sensor transmits measurement data using itssensor radio transmitter/receiver to the at least one hub radiotransmitter/receiver, and receives signals from the at least one hubradio transmitter/receiver; and wherein the at least one hub radiotransmitter/receiver transmits signals to the sensors, receives themeasurement data transmitted by the sensors, and transmits themeasurement data to the system radio transmitter/receiver.
 2. The systemof claim 1, wherein at least one of the plurality of sensors attached tothe first user has a wired connection with the respective hub radiotransmitter/receiver of the first user.
 3. A system for transmittingsensor measurement data from a plurality of users within a simulatedtraining environment comprising: a plurality of sensors attachable tothe plurality of users, each sensor comprising a battery powered sensorradio transmitter/receiver and an antenna, wherein each sensor iscapable of measuring at least one of orientation, position and motion;at least one hub radio transmitter/receiver; and a system radiotransmitter/receiver; wherein each sensor transmits measurement datausing its sensor radio transmitter/receiver to the at least one hubradio transmitter/receiver, and receives signals from the at least onehub radio transmitter/receiver; wherein the at least one hub radiotransmitter/receiver transmits signals to the sensors, receives themeasurement data transmitted by the sensors, and transmits themeasurement data to the system radio transmitter/receiver; and whereinat least one of the plurality of sensors is attachable to a leg ofapparel worn one of the plurality of users, and transmits bothorientation and position information to the at least one hub radiotransmitter/receiver.
 4. The system of claim 3, wherein the sensorattachable to the leg of the apparel worn by the one of the plurality ofusers comprises an orientation indicator and is configured to beinserted into a pocket of the apparel worn by the one of the pluralityof users.